An Empirical Investigation into the Failure Modes of Logistic Regression and Targeted Mitigation Strategies

Abstract

Despite the growing dominance of deep neural architectures, logistic regression continues to hold its ground as a go-to tool for binary classification — largely because of how easy it is to interpret and how cheaply it runs. That said, it is far from bulletproof. Certain data conditions can quietly erode its performance in serious ways. This paper takes a hands-on, experiment-driven approach to investigating four well-known but often under-examined failure scenarios: multicollinearity among input features, decision boundaries that curve rather than cut straight, the convergence breakdown that comes with complete separation in high-dimensional spaces, and the silent damage caused by heavily lopsided class distributions. We ran experiments on purpose-built synthetic datasets as well as two widely used real-world benchmarks from the UCI repository — the Wisconsin Breast Cancer diagnostic set and the Credit Card Fraud Detection collection — measuring outcomes through ROC-AUC and F1-score. On top of documenting where things fall apart, we also tested a suite of targeted fixes: L1 and L2 penalization, polynomial feature expansion, and the SMOTE oversampling method. The numbers paint a clear picture. Without any intervention, logistic regression barely beats a coin flip on non-linear data (ROC-AUC of 0.52) and essentially ignores the minority class under extreme skew (F1 below 0.10). But each of the tested remedies brought meaningful recovery — polynomial expansion pushed non-linear classification up to 0.97 ROC-AUC, while SMOTE lifted minority-class F1 to 0.82. We distill these findings into a straightforward decision guide that practitioners can use to quickly diagnose what is going wrong with their logistic regression model and choose the right corrective action.

Introduction

Although advanced machine learning techniques such as transformer models and deep neural networks dominate modern discussions, logistic regression (LR) remains widely used in practical applications because of its simplicity, interpretability, and reliability. In sensitive fields such as healthcare, finance, and legal decision-making, organizations often require models whose decisions can be explained and justified. Logistic regression provides understandable coefficients, probability estimates, and efficient deployment with limited computational resources.

Logistic regression predicts the probability of a binary outcome by applying the sigmoid function to a weighted combination of input features. The model relies on several important assumptions: predictors should not have strong correlations, the relationship between predictors and the log-odds of the outcome should be approximately linear, and both outcome classes should contain sufficient examples. However, real-world datasets frequently violate these assumptions, causing problems such as unstable coefficients, poor predictions, convergence failures, and unreliable decision-making.

This study investigates when logistic regression becomes unreliable under different types of data degradation and evaluates methods for improving performance. The research focuses on four major failure conditions: multicollinearity, non-linear decision boundaries, complete separation/high dimensionality, and class imbalance. It also compares corrective strategies, including L1 and L2 regularization, polynomial feature transformation, and SMOTE-based synthetic resampling.

The literature review highlights the historical development of logistic regression and its limitations. Previous research identified multicollinearity as a major problem that increases coefficient uncertainty, non-linear relationships as a limitation of linear decision boundaries, and complete separation as a condition where maximum likelihood estimation fails. Researchers have proposed several solutions, including Ridge and Lasso regularization, Firth’s bias reduction methods, polynomial transformations, and SMOTE for handling imbalanced datasets. However, previous studies typically examined these problems separately rather than providing a unified experimental comparison.

The proposed research addresses this gap by using controlled experiments and real-world datasets to measure the impact of different failure conditions. Synthetic datasets are created to simulate:

• Multicollinearity, by increasing feature correlations and observing changes in variance inflation factor (VIF) and coefficient stability.
• Non-linear relationships, using crescent-shaped datasets that cannot be separated by a straight decision boundary.
• Class imbalance, by creating datasets with increasingly unequal class distributions such as 90:10, 95:5, and 99:1 ratios.

The study also validates findings using real-world datasets, including the Breast Cancer Wisconsin Diagnostic dataset and the Credit Card Fraud Detection dataset. Model performance is evaluated using metrics such as precision, recall, F1-score, and ROC-AUC, rather than relying only on accuracy, which can be misleading for imbalanced problems.

The results demonstrate that logistic regression performance declines significantly when its assumptions are violated. In the case of multicollinearity, increasing feature correlation causes coefficient instability, with standard errors increasing dramatically even when predictive performance decreases only slightly. Regularization methods successfully address this issue: Ridge regression improves coefficient stability, while Lasso additionally performs feature selection.

For non-linear decision boundaries, standard logistic regression performs poorly because it can only create linear separation boundaries. On crescent-shaped datasets, baseline logistic regression performs almost randomly, but polynomial feature expansion significantly improves performance by allowing the model to capture curved relationships. Higher-degree polynomial transformations achieve substantially better classification results.

For class imbalance, the research shows that accuracy can provide a misleading impression of model quality. Under extreme imbalance, such as a 99:1 ratio, logistic regression achieves high accuracy by predicting mostly the majority class while failing to identify important minority cases. Applying SMOTE improves minority-class detection by increasing recall, F1-score, and ROC-AUC, although overall accuracy may decrease slightly.

Conclusion

This study set out to put concrete numbers on a question that many practitioners have grappled with intuitively: when exactly does logistic regression break, and what can you do about it? After running a controlled battery of experiments across both synthetic and real-world data, several clear takeaways emerge. Multicollinearity erodes model reliability in a way that is easy to miss if you only watch aggregate prediction metrics. Once the Pearson correlation between features crosses the 0.85 mark (VIF above 7.7), the coefficient standard errors balloon to a point where individual feature interpretations become essentially untrustworthy. Ridge regularization is the cleanest fix, cutting those standard errors by roughly 80% while keeping all features in the model. Non-linearity is, hands down, the most devastating failure scenario we tested. When the real class boundary is curved — as in the crescent-moon pattern — an unmodified logistic regression model performs no better than random guessing, with an ROC-AUC of just 0.52. Polynomial feature engineering at degree 3 pulled that all the way up to 0.97, proving that you do not necessarily need a fancy non-linear classifier — sometimes you just need to give the linear model a richer set of features to work with. Class imbalance creates a mirage of strong performance. A model that achieves 99% accuracy while detecting only 8% of the cases you actually care about is worse than useless — it is dangerous, because it creates false confidence. SMOTE proved to be an effective antidote, boosting minority-class recall from 0.34 to 0.81 at the 95:5 ratio and lifting the ROC-AUC from 0.82 to 0.93. The real-world experiments on the breast cancer and credit card fraud datasets confirmed every major pattern we observed in the synthetic setting. Particularly encouraging was the fraud detection result, where layering SMOTE and L2 regularization together yielded the strongest overall performance (F1 of 0.84, ROC-AUC of 0.98), showing that these fixes compose well. The practical upshot of all this is captured in the decision matrix (Table 8). Before abandoning logistic regression because it is not working, practitioners should first diagnose which specific data pathology is causing the problem and apply the corresponding targeted fix. More often than not, a carefully patched logistic regression model can deliver results that rival far more complex alternatives — while keeping the transparency, speed, and simplicity that made it the right choice in the first place.

References

[1] D. W. Hosmer, S. Lemeshow, and R. X. Sturdivant, Applied Logistic Regression, 3rd ed. Hoboken, NJ, USA: John Wiley & Sons, 2013. [2] A. Agresti, Categorical Data Analysis, 3rd ed. Hoboken, NJ, USA: John Wiley & Sons, 2013. [3] D. R. Cox, \"The regression analysis of binary sequences,\" Journal of the Royal Statistical Society: Series B, vol. 20, no. 2, pp. 215-242, 1958. [4] D. W. Hosmer and S. Lemeshow, Applied Logistic Regression, 2nd ed. New York, NY, USA: John Wiley & Sons, 2000. [5] D. A. Belsley, E. Kuh, and R. E. Welsch, Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York, NY, USA: John Wiley & Sons, 1980. [6] S. Menard, \"Coefficients of determination for multiple logistic regression analysis,\" The American Statistician, vol. 54, no. 1, pp. 17-24, 2000. [7] C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY, USA: Springer, 2006. [8] A. Albert and J. A. Anderson, \"On the existence of maximum likelihood estimates in logistic regression models,\" Biometrika, vol. 71, no. 1, pp. 1-10, 1984. [9] G. Heinze and M. Schemper, \"A solution to the problem of separation in logistic regression,\" Statistics in Medicine, vol. 21, no. 16, pp. 2409-2419, 2002. [10] W. W. Hauck and A. Donner, \"Wald\'s test as applied to hypotheses in logit analysis,\" Journal of the American Statistical Association, vol. 72, no. 360a, pp. 851-853, 1977. [11] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, \"SMOTE: Synthetic Minority Over-sampling Technique,\" Journal of Artificial Intelligence Research, vol. 16, pp. 321-357, 2002. [12] H. He and E. A. Garcia, \"Learning from imbalanced data,\" IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263-1284, 2009. [13] R. Tibshirani, \"Regression shrinkage and selection via the lasso,\" Journal of the Royal Statistical Society: Series B, vol. 58, no. 1, pp. 267-288, 1996. [14] A. E. Hoerl and R. W. Kennard, \"Ridge regression: Biased estimation for nonorthogonal problems,\" Technometrics, vol. 12, no. 1, pp. 55-67, 1970. [15] H. Zou and T. Hastie, \"Regularization and variable selection via the elastic net,\" Journal of the Royal Statistical Society: Series B, vol. 67, no. 2, pp. 301-320, 2005. [16] F. Pedregosa et al., \"Scikit-learn: Machine learning in Python,\" Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011. [17] G. Lemaitre, F. Nogueira, and C. K. Aridas, \"Imbalanced-learn: A Python toolbox to tackle the curse of imbalanced datasets in machine learning,\" Journal of Machine Learning Research, vol. 18, no. 17, pp. 1-5, 2017. [18] W. N. Street, W. H. Wolberg, and O. L. Mangasarian, \"Nuclear feature extraction for breast tumor diagnosis,\" in Proc. IS&T/SPIE Int. Symp. Electronic Imaging: Science and Technology, vol. 1905, pp. 861-870, 1993. [19] A. Dal Pozzolo, O. Caelen, R. A. Johnson, and G. Bontempi, \"Calibrating probability with undersampling for unbalanced classification,\" in Proc. IEEE Symp. Computational Intelligence and Data Mining (CIDM), pp. 159-166, 2015.

Copyright

Copyright © 2026 Afaan Hussain Shah, Saqib Anjum Masoodi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.